Reading Articles

Neil Lund

2024-09-12

How to read an academic article

  • Throughout this class, you’ll periodically be asked to read academic articles on a topic that have empirical research.

  • Sometimes these will contain methods you’ve never encountered before[^1] so its important to know how to get useful information in these situations

(This is true for academics too. There’s a lot of methods out there, and most of us only learn stuff that’s relevant for our own research or teaching)

Parts of an empirical journal article

  • Abstract: summarizes the following sections in a few sentences

  • Introduction: lays out the research question in more detail

    • This section will usually include a literature review that discusses existing findings

    • It may include a separate theory section that details how the authors’ approach differs from prior work or the limits of existing knowledge on a topic

  • Methods: talks about the data collection strategy, measurement of the variables, and statistical models

  • Findings: shows the results of the statistical tests. Might include multiple robustness checks to anticipate potential critiques of the main result

  • Discussion/Conclusion: summarizes the findings again, discusses implications, and future work

What actually matters?

You’ll find a lot of things in methods sections, but not all of them are equally important.

What actually matters?

You’ll find a lot of things in methods sections, but not all of them are equally important.

  1. The outcome being explained (Y)

  2. The explanatory variables (X)

  3. The relationship between X and Y

  4. The measurement, data, and model

  5. The alternative explanations

What actually matters?

You’ll find a lot of things in methods sections, but not all of them are equally important.

1. The outcome being explained (Y)

2. The explanatory variables (X)

3. The relationship between X and Y

  1. The measurement, data, and model

  2. The alternative explanations

1-3 are crucial.

4 and 5 are important, but only insomuch as they might lead you to doubt 1-3

What to understand first

We advance social movement and diffusion theories by exploring the role of online activities in the spread of the Occupy Wall Street movement. The results from event history analyses suggest that, after controlling for community characteristics, online activities on Facebook and Twitter are associated with the spread of protests.

From Vasi, Ion Bogdan, and Chan S. Suh. “Online activities, spatial proximity, and the diffusion of the Occupy Wall Street movement in the United States.”Mobilization: An International Quarterly21.2 (2016): 139-154. https://meridian.allenpress.com/mobilization/article/21/2/139/82998/Online-Activities-Spatial-Proximity-and-the

What to understand first: Y

What is the research question, what outcome is being explained? (DV)

We advance social movement and diffusion theories by exploring the role of online activities in the spread of the Occupy Wall Street movement. The results from event history analyses suggest that, after controlling for community characteristics, online activities on Facebook and Twitter are associated with the spread of protests.

What to understand first: X

What factors are predicting/causing the outcome?

We advance social movement and diffusion theories by exploring the role of online activities in the spread of the Occupy Wall Street movement. The results from event history analyses suggest that, after controlling for community characteristics, online activities on Facebook and Twitter are associated with the spread of protests.

What to understand first: how X relates to Y

Is the IV having a positive or negative effect? Or maybe curvilinear or conditional?

We advance social movement and diffusion theories by exploring the role of online activities in the spread of the Occupy Wall Street movement. The results from event history analyses suggest that, after controlling for community characteristics, online activities on Facebook and Twitter are associated with the spread of protests.

What to understand first: the reason for the relationship

  • An explanation for why these things are related should be laid out in the theory section if not the abstract.

    • Activists use social media to “rapidly disseminate information about grievances and to organize offline events.”

  • Once you’ve got this down, you can start thinking about potential problems.

What are the potential problems?

Issue/error type Minor issues Major issues
Measurement Lost forms, instrument failure, data corruption Lack of construct validity, researcher bias
Sampling/uncertainty Lucky/unlucky draws. Insufficient data. Differential non-response, attrition, p-hacking
Model design problems Heteroskedasticity, autocorrelation, multicollinearity Omitted variable bias/spurious correlation

In general: things that cause random “noise” are very easy to deal with by collecting more data or changing our models, but things that cause bias are more difficult.

Example

Skipping to the end

Why are there four different models here?? What are linear panel regressions? What is a robust standard error?

  • Regardless of the model: positive sign means positive relationship, Negative sign means a negative relationship. Asterisks mean that the result is unlikely to be a product of random sampling error (but its still possible!)

  • the boring truth is that there are four different models here mostly to convince people that the authors tested the same thing four different ways and got similar results.

  • The article text is far more useful for getting a sense of the findings.

The main variables

What is the outcome the authors are trying to explain?

The dependent variable here is “democratization” and the primary independent variables are “mobilization for autocracy” and “mobilization for democracy”.

The proposed mechanism

  • The proposed relationship
    • “…we argue that mobilization in support for democracy may enhance democratization and democratic stability while mobilization for autocracy has the opposite effects” (p. 6)
  • The theoretical link

    • “successful protest has the possibility to expand opposition to the incumbent by revealing new information to others”

    • “reform as a response to popular disquiet may become the more attractive option for the incumbents [compared to repression]”

    • “counterelites may become emboldened…if collective action reveals greater disquiet with incumbent rule.”

  • The hypotheses:
    • Mobilization in support of democracy –> higher level of democracy
    • During liberalization, mobilization increases probability of successful transition

The measurements: mobilization

“surveyed experts and asked them to estimate the size and frequency of mass mobilization”

The measurements: democracy

“Electoral democracy index” (also created from expert surveys)

Initial Results

Controls and modeling strategies

  • What other factors might explain the observed relationship here?

  • Specifically, you are looking for things that might be correlated with both the IV and the DV

  • what happens if I measure the same country year after year?

Conclusion section

(you don’t have to take their word for it, but good papers will usually just tell you what they think the results mean)

Back to the results

  • Model coefficients tell you the effect of a variable on the outcome
  • p.values and standard errors tell you about the likelihood that a result is attributable to random chance. Less than .05 is considered “statistically significant”, but this is less meaningful than it sounds.
  • Control variables attempt to account for spurious correlations (but we can’t control for everything)
  • Measures like Adj. R^2 can give you a rough sense of model “fit”.

What actually matters?

  • Try to identify the research question, DV, IV, and the proposed relationship first. (this should all be clear before you get to the methods section)

    • There can be multiple outcomes or causes, and the proposed relationship could be complex. See if you can draw it on a chart, or lay it out in a table.

  • Read the abstract closely - a good one will preview the main findings.

  • Try to get a sense of what parts are common practice as opposed to central to the findings. (if the author doesn’t elaborate on why they chose a particular measure or method or what it means, then it is probably okay to ignore it)

  • Good papers will try to give a sense of the size of an effect and its practical importance, rather than just showing a bunch of coefficients.